SWIFT Flow - Sparsely Weighted SIFT Matches based Optical Flow

1. Problem Description The problem of optical flow estimation focuses on generating the pattern of (apparent) motion of objects, surfaces, and edges in a visual scene caused by the relative motion between the camera and the scene. Sequences of ordered images allow the estimation of motion as either instantaneous image velocities or discrete image displacements. The optical flow estimation methods try to calculate the motion of each pixel (denoted by a direction and scale) between two image frames which are taken at some time instant t and the next time instant t+ 1. Optical flow is widely used in computer vision, with applications in tracking/motion detection, object segmentation, time-to-collision and focus of expansion calculations, motion compensated encoding, and stereo disparity measurement. Optical flow has wide applications in mobile robotics and egocentric-motion where the flow vector is used to perform visual odometry which is used to estimate the displacement, speed and other parameters of the robot. However, calculating the flow in real world images is a challenging task in computer vision. The large disparities in depth in real world images causing sudden changes in the flow direction at occlusion boundaries, regions with only a little or no texture and changes in illumination and perspective are some of the challenges in optical flow estimation. Moreover, optical flow estimation needs to be real-time to be usable for motion in robotics. 2. Related Work Optical flow estimation has been extensively studied in literature with most methods categorized into local or global methods. One of the most popular local methods is the Lucas-Kanade image registration method [9]. This method works by looking for a match for each pixel within a small neighborhood of pixels in the next image. To speed up the process, the iterative Lucas-Kanade method (and most other similar local methods) employ a multi-resolution estimation scheme by constructing an image pyramid and searching for matches in a decimated version of the two images and then propagating this information to the lower levels in the pyramid. Examples of global methods for optical flow estimation methods include the Horn-Schunck method [6] which optimizes a function based on residuals from the brightness constancy constraint and a particular regularization term expressing the expected smoothness of the flow field, and the Buxton-Buxton method [2] based on a model of the motion of edges in image sequences. Local methods are robust to noise but are generally unable to produce dense optical flows; global methods produce dense optical flows but are empirically known to be sensitive to noise. Recently, Liu et al. [7] proposed SIFT-flow – a local method which computes the flow for each pixel. However, an important step in their method is that every pixel in the image is represented by a 128-dimensional SIFT feature [8] to produce a SIFT-image (instead of the usual pixel-intensity) and the optical flow is estimated between the SIFT images. As expected, SIFT-flow is extremely slow but much more robust and accurate than the regular LucasKanade style methods. However, this method does not use the power of RANSAC [3] to efficiently obtain accurate matches to help predict the flow vector. We believe that RANSAC based SIFT matching can be used to obtain accurate flow estimates with lower computational costs [4]. 3. Approach To estimate the optical flow between two frames Ft and Ft+1, we begin by obtaining sparse correspondences between the two frames by matching the respective SIFT vectors. Unlike Liu et al. [7], we compute the SIFT vectors only at the Harris-corner [5] interest points in the image. We use RANSAC[3] to remove outliers from the set of sparse matches. This process can be performed more efficiently than matching the SIFT features at every pixel. This will provide us with sparse correspondences between the two frames. The sparse SIFT matches provides us with the optical flow at these feature points. The (x,y) location of the matches between the two images can be used to compute the flow vectors at these points. This set of sparse flow vectors at the interest points is the seed for the prediction step. Using the flow at these seed locations, we use an iterative algorithm, SWIFT-flow (Algorithm 3), to compute the optical flow for the entire image. SWIFT-flow begins with a set S of seed locations with their optical flows which is obtained from the SIFT matches and iteratively performs the following two steps until no more progress is made: